1. Field of the Invention
The present invention relates generally to image sensors, and specifically to a method for correcting column fixed pattern noise (FPN) in CMOS image sensors.
2. Description of the Related Art
Visible imaging systems implemented using CMOS image sensors reduce camera noise, cost and power while simultaneously improving resolution and capture rate. The most advanced and highest performance cameras use CMOS imaging System-on-Chip (iSoC) sensors that efficiently couple low-noise image detection and processing by using various supporting blocks integrated on a single chip.
On the other hand, CMOS iSoC's are prone to producing image artifacts that are by-products of the specific analog readout architecture used to capture the image. A common example of one such image artifact is column Fixed Pattern Noise (FPN), which arises when each column of pixels has a different fixed offset. When the peak-to-peak variation of such offsets is sufficiently high so as to make the columns visible in the still or video image, steps must be taken to correct it since column FPN can be most evident at nominal gain. Other artifacts include row FPN and pixel-to-pixel FPN; these can be similarly generated by amplifier offsets, but can also be created by the process of sampling reference noise at different times for each pixel or row. Pixel-to-pixel and row FPN are particularly evident in low light conditions wherein maximum sensor gain is necessary.
Various techniques are used to correct FPN in both the analog circuits located throughout the sensor's signal chain and in the downstream digital circuitry. In the latter case, the downstream circuits can be on-chip or off-chip, i.e., in the downstream camera electronics. An early example of an image sensor that used an external differential amplifier to subtract the stored FPN data from the two-dimensional sensor's video output is taught in U.S. Pat. No. 3,067,283.
Other prior art instead integrated FPN suppression in the iSoC by either analog or digital means. U.S. Pat. No. 5,892,540, for example, teaches a self-correcting column buffer that actively suppresses column offsets in the analog domain to the order of tens of microvolts as each pixel is read out at each column. While this methodology corrects each column buffer's dc offset, it does not correct offsets generated later in the signal processing chain. Nevertheless, the '540 patent improved on earlier FPN compensation methods that first determined FPN in the absence of the light, stored the offset terms, and then compensated for FPN while generating the video stream, such as both the '283 patent and U.S. Pat. No. 3,949,162. The '162 patent corrects the offsets in the analog domain after digitally acquiring the data.
It was determined that it is necessary to include dedicated optically black (OB) pixels (pixels shielded from light pickup) in the image sensor and that these should be located in the periphery surrounding the light-sensing area. OB pixels are useful to optimize black clamping and to properly facilitate fixed pattern noise compensation. U.S. Pat. No. 4,678,938 hence teaches column-wise and row-wise embodiments for reading OB pixels in a feedback-controlled manner to dynamically eliminate offsets in each column or row. U.S. Pat. No. 4,839,729 later improves the efficacy of the '938 patent by instead reading each line of active video at the same time as a stored line of OB video and using the differential amplifier scheme pioneered in the '283 patent to eliminate FPN. U.S. Pat. No. 6,788,340, later combined the various cited and other prior art to teach an integrated solution that could be included in a single iSoC. The '340 patent specifically combines optically black pixels, which are located at the periphery of the image sensor, with a digital controller and differential programmable gain amplifier 24. Nevertheless, the '340 patent does not teach specific means and an effective algorithm for correcting FPN. Instead, the focus of the '340 patent is to enable dynamic adjustment of the video to always use the largest possible ADC range and to optimize image brightness. U.S. Pat. No. 7,098,950 later improved on black clamp operation by not including defective pixels.
Column Fixed Pattern Noise offsets are determined in the present invention by estimating the specific column FPN using a set of optically black calibration pixels (OB pixels) within each column. However, there are several factors that can influence the effectiveness of FPN determination. Factors degrading efficacy include the amount of temporal noise in the OB pixels, the presence of random individual pixel offsets (pixel FPN), the quantity of available OB pixels per column, and the digital precision of the correction. These factors are mitigated by the present invention.
In calculating the column offsets, the influence of pixel temporal noise is eliminated by using a push-up/push-down approach in which small incremental adjustments are continuously made. Depending on the magnitude of the adjustment, this offers the most precise offset calculation, since the temporal noise is effectively filtered out. However, depending on how large the offset variations are, and on the number of available OB samples there are in each frame, it can take a long time (many frames) for this method to converge. Furthermore, video imaging systems usually incorporate some kind of dynamic gain adjustment so that the response of the sensor can be continuously adjusted to suit the scene conditions. The calculated column offsets may, depending on the readout architecture, become invalid by such gain adjustments, thereby requiring continual re-calculation.
In order to have a correction value that is always valid, an alternative approach is to take an average of the OB pixels within each column and perform a new calculation for each frame. In this case, the temporal noise has more influence on the precision of the calculation. Furthermore, the mean may be adversely affected by a few outlier pixels, i.e. pixels not falling within a Gaussian distribution of pixel offset, or those not adhering to a Gaussian temporal distribution themselves.
Understanding these drawbacks and limitations, the present invention teaches a method for quickly converging to the correct offsets to eliminate column FPN in the presence of other pattern noise, temporal noise, and flicker noise.